Instrument shape determination

ABSTRACT

A method of controlling an instrument involves measuring strain of one or more optical fibers associated with an elongate shaft of an instrument, determining a first shape estimation of the elongate shaft based on the measured strain, determining, based on the measured strain, a strain-indicated characteristic of the elongate shaft with respect to a mechanical condition of the elongate shaft, determining an expected range of the mechanical condition of the elongate shaft, determining that the strain-indicated characteristic is outside of the expected range of the mechanical condition, and in response to the determination that the strain-indicated characteristic is outside of the expected range, determining a second shape estimation of the elongate shaft, the second shape estimation being based to a lesser degree on the measured strain compared to the first shape estimation.

CROSS-REFERENCE TO RELATED APPLICATION

This application is a continuation of U.S. patent application Ser. No.17/124,151, filed Dec. 16, 2020, which is a continuation of U.S. patentapplication Ser. No. 16/531,580, filed Aug. 5, 2019, which claims thebenefit of U.S. Provisional Application No. 62/715,668, filed Aug. 7,2018, the complete disclosures of which are hereby incorporated byreference in their entirety.

TECHNICAL FIELD

The systems and methods disclosed herein are directed to surgicalrobotics, and more particularly to navigation of a medical instrumentwithin a tubular network of a patient's body.

BACKGROUND

Bronchoscopy is a medical procedure that allows a physician to examinethe inside conditions of a patient's lung airways, such as bronchi andbronchioles. The lung airways carry air from the trachea, or windpipe,to the lungs. During the medical procedure, a thin, flexible tubulartool, known as a bronchoscope, may be inserted into the patient's mouthand passed down the patient's throat into his/her lung airways, andpatients are generally anesthetized in order to relax their throats andlung cavities for surgical examinations and operations during themedical procedure.

A bronchoscope can include a light source and a small camera that allowsa physician to inspect a patient's windpipe and airways, and a rigidtube may be used in conjunction with the bronchoscope for surgicalpurposes, e.g., when there is a significant amount of bleeding in thelungs of the patient or when a large object obstructs the throat of thepatient. When the rigid tube is used, the patient is often anesthetized.Robotic bronchoscopes provide tremendous advantages in navigationthrough tubular networks. They can ease use and allow therapies andbiopsies to be administered conveniently even during the bronchoscopystage.

Apart from mechanical devices or platforms, e.g., robotic bronchoscopesdescribed above, various methods and software models may be used to helpwith the surgical operations. As an example, a computerized tomography(CT) scan of the patient's lungs is often performed during pre-operationof a surgical examination. Data from the CT scan may be used to generatea three-dimensional (3D) model of airways of the patient's lungs, andthe generated 3D model enables a physician to access a visual referencethat may be useful during the operative procedure of the surgicalexamination.

However, previous techniques for navigation of tubular networks stillhave challenges, even when employing medical devices (e.g., roboticbronchoscopes) and when using existing methods (e.g., performing CTscans and generating 3D models). As one example, motion estimation of amedical device (e.g., a bronchoscope tool) inside a patient's body maynot be accurate based on location and orientation change of the device,and as a result the device's position may not be accurately or correctlylocalized inside the patient's body in real time. Inaccurate locationinformation for such an instrument may provide misleading information tothe physician that uses the 3D model as a visual reference duringmedical operation procedures.

Thus, there is a need for improved techniques for navigating through anetwork of tubular structures.

SUMMARY

Robotic systems and methods for navigation of luminal network that canimprove strain-based shape sensing are described. In one aspect, thesystem can compare strain-based shape data to shape data determinedbased on robotic data (e.g., command data, force and distance data,mechanical model data, kinematic model data, etc.) and adjust thestrain-based shape data as necessary. Any portion of the strain-basedshape data can be adjusted, weighted differently, or discarded based onthe comparison. For example, data from trustworthy sources may indicatethat the shape of an instrument exhibits or should exhibit one or morecharacteristics. If the system determines that any portion of thestrain-based shape data is not in agreement with such characteristics,the system may adjust the portion of the strain-based shape data suchthat the adjusted strain-based shape data is in agreement with thecharacteristics of the instrument.

Accordingly, one aspect relates to a method of navigating an instrumentwithin an interior region of a body. The method may include: accessingrobotic data regarding the instrument; accessing strain data from anoptical fiber positioned within the instrument that is indicative of astrain on a portion of the instrument positioned within the interiorregion of the body; determining shape data based on the strain data;comparing the robotic data and the shape data; adjusting the shape databased on the comparison of the robotic data and the shape data;determining an estimated state of the instrument based on the adjustedshape data; and outputting the estimated state of the instrument.

The aspect described in the above paragraph may also include one or moreof the following features in any combination: (a) wherein adjusting theshape data comprises modifying at least a portion of the shape data suchthat the determination of the estimated state of the instrument is basedon the modified portion of the shape data; (b) wherein adjusting theshape data comprises removing at least a portion of the shape data suchthat the determination of the estimated state of the instrument is notbased on the removed portion of the shape data; (c) wherein the methodfurther includes accessing electromagnetic (EM) data captured using (i)an EM sensor located proximal to a tip of the instrument and (ii) atleast one external EM sensor or EM field generator located external tothe body, comparing the EM data and the shape data, and furtheradjusting the shape data based on the comparison of the EM data and theshape data; (d) wherein the method further includes accessing image datacaptured by an imaging device located proximal to a tip of theinstrument, comparing the image data and the shape data, and furtheradjusting the shape data based on the comparison of the image data andthe shape data; (e) wherein the strain data is generated based on fiberBragg gratings (FBGs) created on a portion of the optical fiber; (f)wherein the shape data comprises one of a curvature value of the portionof the instrument or time history data of the portion of the instrument;(g) wherein the method further includes adjusting the shape data basedon a determination that the curvature value is greater than or equal toa threshold curvature value in the robotic data; (h) wherein the methodfurther includes adjusting the shape data based on a determination thatthe time history data satisfies a threshold time history condition inthe robotic data; (i) wherein the method further includes adjusting theshape data based on a change of temperature; (j) wherein the methodfurther includes adjusting the shape data based on a determination thata tip of the instrument is being articulated; (k) wherein the methodfurther includes adjusting the shape data based on a determination thatnon-shape-changing strain is being applied to the instrument; (l)wherein the method further includes assigning, based on a determinationthat a first portion of the instrument comprises a distal end of theinstrument, a confidence value to the robotic data corresponding to thefirst portion that is higher than that assigned to the shape datacorresponding to the first portion; (m) wherein the method furtherincludes assigning, based on a determination that a first portion of theinstrument comprises a proximal end of the instrument, a confidencevalue to the robotic data corresponding to the first portion that islower than that assigned to the shape data corresponding to the firstportion; (n) wherein the method further includes determining anestimated state of a sheath covering the instrument based on theestimated state of the instrument; (o) wherein the method furtherincludes assigning a confidence value to the shape data based on acomparison of the shape data and additional data indicative of a shapeof a sheath covering the instrument; (p) wherein the method furtherincludes determining, based on the estimated state of the instrument,that a damage to the instrument is imminent, and controlling theinstrument such that the damage is avoided; and (q) wherein the methodfurther includes determining that a mismatch between the robotic dataand the shape data has been detected for at least a threshold amount oftime, and outputting an alert indicating that the instrument may bedamaged.

Another aspect relates to a method of navigating an instrument within aninterior region of a body. The method may include: accessing roboticdata regarding the instrument; accessing strain data from an opticalfiber positioned within the instrument that is indicative of a strain ona portion of the instrument positioned within the interior region of thebody; determining shape data based on the strain data; comparing therobotic data and the shape data; adjusting a confidence value associatedwith the shape data based on the comparison of the robotic data and theshape data; determining an estimated state of the instrument based onthe adjusted confidence value; and outputting the estimated state of theinstrument.

The aspect described in the above paragraph may also include one or moreof the following features in any combination: (a) wherein the methodfurther includes accessing electromagnetic (EM) data captured using (i)an EM sensor located proximal to a tip of the instrument and (ii) atleast one external EM sensor or EM field generator located external tothe body, comparing the EM data and the shape data, and adjusting theconfidence value associated with the shape data based further on thecomparison of the EM data and the shape data; (b) wherein the methodfurther includes accessing image data captured by an imaging devicelocated proximal to a tip of the instrument, comparing the image dataand the shape data, and adjusting the confidence value associated withthe shape data based further on the comparison of the image data and theshape data; (c) wherein the strain data is generated based on fiberBragg gratings (FBGs) created on a portion of the optical fiber; (d)wherein the shape data comprises one of a curvature value of the portionof the instrument or time history data of the portion of the instrument;(e) wherein the method further includes adjusting the confidence valuebased on a determination that the curvature value is greater than orequal to a threshold curvature value in the robotic data; (f) whereinthe method further includes adjusting the confidence value based on adetermination that the time history data satisfies a threshold timehistory condition in the robotic data; (g) wherein the method furtherincludes adjusting the confidence value based on a change oftemperature; (h) adjusting the confidence value based on a determinationthat a tip of the instrument is being articulated; (i) wherein themethod further includes adjusting the confidence value based on adetermination that non-shape-changing strain is being applied to theinstrument; (j) wherein the method further includes assigning, based ona determination that a first portion of the instrument comprises adistal end of the instrument, a confidence value to the robotic datacorresponding to the first portion that is higher than that assigned tothe shape data corresponding to the first portion; (k) wherein themethod further includes assigning, based on a determination that a firstportion of the instrument comprises a proximal end of the instrument, aconfidence value to the robotic data corresponding to the first portionthat is lower than that assigned to the shape data corresponding to thefirst portion; (l) wherein the method further includes determining anestimated state of a sheath covering the instrument based on theestimated state of the instrument; (m) wherein the method furtherincludes adjusting the confidence value based further on a comparison ofthe shape data and additional data indicative of a shape of a sheathcovering the instrument; (n) wherein the method further includesdetermining, based on the estimated state of the instrument, that adamage to the instrument is imminent, and controlling the instrumentsuch that the damage is avoided; and (o) wherein the method furtherincludes determining that a mismatch between the robotic data and theshape data has been detected for at least a threshold amount of time,and outputting an alert indicating that the instrument may be damaged.

BRIEF DESCRIPTION OF THE DRAWINGS

The disclosed aspects will hereinafter be described in conjunction withthe appended drawings, provided to illustrate and not to limit thedisclosed aspects, wherein like designations denote like elements.

FIG. 1A shows an example surgical robotic system, according to oneembodiment.

FIGS. 1B-1F show various perspective views of a robotic platform coupledto the surgical robotic system shown in FIG. 1A, according to oneembodiment.

FIG. 2 shows an example command console for the example surgical roboticsystem, according to one embodiment.

FIG. 3A shows an isometric view of an example independent drivemechanism of the instrument device manipulator (IDM) shown in FIG. 1A,according to one embodiment.

FIG. 3B shows a conceptual diagram that shows how forces may be measuredby a strain gauge of the independent drive mechanism shown in FIG. 3A,according to one embodiment.

FIG. 4A shows a top view of an example endoscope, according to oneembodiment.

FIG. 4B shows an example endoscope cross-section of the endoscope shownin FIG. 4A, according to one embodiment.

FIG. 4C shows an example strain-based shape sensing mechanism, accordingto one embodiment.

FIGS. 4D-4E show actual shapes of an example endoscope and strain-basedpredictions of the endoscope, according to one embodiment.

FIG. 5 shows an example schematic setup of an EM tracking systemincluded in a surgical robotic system, according to one embodiment.

FIGS. 6A-6B show an example anatomical lumen and an example 3D model ofthe anatomical lumen, according to one embodiment.

FIG. 7 shows a computer-generated 3D model representing an anatomicalspace, according to one embodiment.

FIG. 8A shows a high-level overview of an example block diagram of anavigation configuration system, according to one embodiment.

FIG. 8B shows a block diagram illustrating example modules included inthe strain-based algorithm module, according to one embodiment.

FIG. 8C shows a block diagram illustrating examples of robotic datastored in the robotic data store, according to one embodiment.

FIG. 9 shows an example block diagram of a shape data determinationmodule, according to one embodiment.

FIG. 10 shows an example block diagram of a shape data comparison moduleand a shape data adjustment module, according to one embodiment.

FIG. 11 shows an example block diagram of a shape-based state estimationmodule, according to one embodiment.

FIG. 12 shows a flowchart illustrating an example method operable by asurgical robotic system, or component(s) thereof, for determining andadjusting shape data, according to one embodiment.

FIG. 13 shows a conceptual diagram illustrating an example methodoperable by a surgical robotic system, or component(s) thereof, foroperating an instrument, according to one embodiment.

FIG. 14 shows a conceptual diagram illustrating an example methodoperable by a surgical robotic system, or component(s) thereof, foroperating an instrument, according to one embodiment.

Reference will now be made in detail to several embodiments, examples ofwhich are illustrated in the accompanying figures. It is noted thatwherever practicable similar or like reference numbers may be used inthe figures and may indicate similar or like functionality. The figuresdepict embodiments of the described system (or method) for purposes ofillustration only. One skilled in the art will readily recognize fromthe following description that alternative embodiments of the structuresand methods illustrated herein may be employed without departing fromthe principles described herein.

DETAILED DESCRIPTION I. Surgical Robotic System

FIG. 1A shows an example surgical robotic system 100, according to oneembodiment. The surgical robotic system 100 includes a base 101 coupledto one or more robotic arms, e.g., robotic arm 102. The base 101 iscommunicatively coupled to a command console, which is further describedwith reference to FIG. 2 in Section II. Command Console. The base 101can be positioned such that the robotic arm 102 has access to perform asurgical procedure on a patient, while a user such as a physician maycontrol the surgical robotic system 100 from the comfort of the commandconsole. In some embodiments, the base 101 may be coupled to a surgicaloperating table or bed for supporting the patient. Though not shown inFIG. 1 for purposes of clarity, the base 101 may include subsystems suchas control electronics, pneumatics, power sources, optical sources, andthe like. The robotic arm 102 includes multiple arm segments 110 coupledat joints 111, which provides the robotic arm 102 multiple degrees offreedom, e.g., seven degrees of freedom corresponding to seven armsegments. The base 101 may contain a source of power 112, pneumaticpressure 113, and control and sensor electronics 114—includingcomponents such as a central processing unit, data bus, controlcircuitry, and memory—and related actuators such as motors to move therobotic arm 102. The electronics 114 in the base 101 may also processand transmit control signals communicated from the command console.

In some embodiments, the base 101 includes wheels 115 to transport thesurgical robotic system 100. Mobility of the surgical robotic system 100helps accommodate space constraints in a surgical operating room as wellas facilitate appropriate positioning and movement of surgicalequipment. Further, the mobility allows the robotic arms 102 to beconfigured such that the robotic arms 102 do not interfere with thepatient, physician, anesthesiologist, or any other equipment. Duringprocedures, a user may control the robotic arms 102 using controldevices such as the command console.

In some embodiments, the robotic arm 102 includes set up joints that usea combination of brakes and counter-balances to maintain a position ofthe robotic arm 102. The counter-balances may include gas springs orcoil springs. The brakes, e.g., fail safe brakes, may be includemechanical and/or electrical components. Further, the robotic arms 102may be gravity-assisted passive support type robotic arms.

Each robotic arm 102 may be coupled to an instrument device manipulator(IDM) 117 using a mechanism changer interface (MCI) 116. The IDM 117 canbe removed and replaced with a different type of IDM, for example, afirst type of IDM manipulates an endoscope, while a second type of IDMmanipulates a laparoscope. The MCI 116 includes connectors to transferpneumatic pressure, electrical power, electrical signals, and opticalsignals from the robotic arm 102 to the IDM 117. The MCI 116 can be aset screw or base plate connector. The IDM 117 manipulates surgicalinstruments such as the endoscope 118 using techniques including directdrive, harmonic drive, geared drives, belts and pulleys, magneticdrives, and the like. The MCI 116 is interchangeable based on the typeof IDM 117 and can be customized for a certain type of surgicalprocedure. The robotic arm 102 can include a joint level torque sensingand a wrist at a distal end, such as the KUKA AG® LBR5 robotic arm.

The endoscope 118 is a tubular and flexible surgical instrument that isinserted into the anatomy of a patient to capture images of the anatomy(e.g., body tissue). In particular, the endoscope 118 includes one ormore imaging devices (e.g., cameras or other types of optical sensors)that capture the images. The imaging devices may include one or moreoptical components such as an optical fiber, fiber array, or lens. Theoptical components move along with the tip of the endoscope 118 suchthat movement of the tip of the endoscope 118 results in changes to theimages captured by the imaging devices. The endoscope 118 is furtherdescribed with reference to FIGS. 3A-4B in Section IV. Endoscope.

Robotic arms 102 of the surgical robotic system 100 manipulate theendoscope 118 using elongate movement members. The elongate movementmembers may include pull wires, also referred to as pull or push wires,cables, fibers, or flexible shafts. For example, the robotic arms 102actuate multiple pull wires coupled to the endoscope 118 to deflect thetip of the endoscope 118. The pull wires may include both metallic andnon-metallic materials such as stainless steel, Kevlar, tungsten, carbonfiber, and the like. The endoscope 118 may exhibit nonlinear behavior inresponse to forces applied by the elongate movement members. Thenonlinear behavior may be based on stiffness and compressibility of theendoscope 118, as well as variability in slack or stiffness betweendifferent elongate movement members.

FIGS. 1B-1F show various perspective views of the surgical roboticsystem 100 coupled to a robotic platform 150 (or surgical bed),according to various embodiments. Specifically, FIG. 1B shows a sideview of the surgical robotic system 100 with the robotic arms 102manipulating the endoscope 118 to insert the endoscope inside apatient's body, and the patient is lying on the robotic platform 150.FIG. 1C shows a top view of the surgical robotic system 100 and therobotic platform 150, and the endoscope 118 manipulated by the roboticarms is inserted inside the patient's body. FIG. 1D shows a perspectiveview of the surgical robotic system 100 and the robotic platform 150,and the endoscope 118 is controlled to be positioned horizontallyparallel with the robotic platform. FIG. 1E shows another perspectiveview of the surgical robotic system 100 and the robotic platform 150,and the endoscope 118 is controlled to be positioned relativelyperpendicular to the robotic platform. In more detail, in FIG. 1E, theangle between the horizontal surface of the robotic platform 150 and theendoscope 118 is 75 degree. FIG. 1F shows the perspective view of thesurgical robotic system 100 and the robotic platform 150 shown in FIG.1E, and in more detail, the angle between the endoscope 118 and thevirtual line 160 connecting one end 180 of the endoscope 118 and therobotic arm 102 that is positioned relatively farther away from therobotic platform is 90 degree.

II. Command Console

FIG. 2 shows an example command console 200 for the example surgicalrobotic system 100, according to one embodiment. The command console 200includes a console base 201, display modules 202, e.g., monitors, andcontrol modules, e.g., a keyboard 203 and joystick 204. In someembodiments, one or more of the command console 200 functionality may beintegrated into a base 101 of the surgical robotic system 100 or anothersystem communicatively coupled to the surgical robotic system 100. Auser 205, e.g., a physician, remotely controls the surgical roboticsystem 100 from an ergonomic position using the command console 200.

The console base 201 may include a central processing unit, a memoryunit, a data bus, and associated data communication ports that areresponsible for interpreting and processing signals such as cameraimagery and tracking sensor data, e.g., from the endoscope 118 shown inFIG. 1 . In some embodiments, both the console base 201 and the base 101perform signal processing for load-balancing. The console base 201 mayalso process commands and instructions provided by the user 205 throughthe control modules 203 and 204. In addition to the keyboard 203 andjoystick 204 shown in FIG. 2 , the control modules may include otherdevices, for example, computer mice, trackpads, trackballs, controlpads, video game controllers, and sensors (e.g., motion sensors orcameras) that capture hand gestures and finger gestures.

The user 205 can control a surgical instrument such as the endoscope 118using the command console 200 in a velocity mode or position controlmode. In velocity mode, the user 205 directly controls pitch and yawmotion of a distal end of the endoscope 118 based on direct manualcontrol using the control modules. For example, movement on the joystick204 may be mapped to yaw and pitch movement in the distal end of theendoscope 118. The joystick 204 can provide haptic feedback to the user205. For example, the joystick 204 vibrates to indicate that theendoscope 118 cannot further translate or rotate in a certain direction.The command console 200 can also provide visual feedback (e.g., pop-upmessages) and/or audio feedback (e.g., beeping) to indicate that theendoscope 118 has reached maximum translation or rotation.

In position control mode, the command console 200 uses athree-dimensional (3D) map of a patient and pre-determined computermodels of the patient to control a surgical instrument, e.g., theendoscope 118. The command console 200 provides control signals torobotic arms 102 of the surgical robotic system 100 to manipulate theendoscope 118 to a target location. Due to the reliance on the 3D map,position control mode requires accurate mapping of the anatomy of thepatient.

In some embodiments, users 205 can manually manipulate robotic arms 102of the surgical robotic system 100 without using the command console200. During setup in a surgical operating room, the users 205 may movethe robotic arms 102, endoscopes 118, and other surgical equipment toaccess a patient. The surgical robotic system 100 may rely on forcefeedback and inertia control from the users 205 to determine appropriateconfiguration of the robotic arms 102 and equipment.

The display modules 202 may include electronic monitors, virtual realityviewing devices, e.g., goggles or glasses, and/or other means of displaydevices. In some embodiments, the display modules 202 are integratedwith the control modules, for example, as a tablet device with atouchscreen. Further, the user 205 can both view data and input commandsto the surgical robotic system 100 using the integrated display modules202 and control modules.

The display modules 202 can display 3D images using a stereoscopicdevice, e.g., a visor or goggle. The 3D images provide an “endo view”(i.e., endoscopic view), which is a computer 3D model illustrating theanatomy of a patient. The “endo view” provides a virtual environment ofthe patient's interior and an expected location of an endoscope 118inside the patient. A user 205 compares the “endo view” model to actualimages captured by a camera to help mentally orient and confirm that theendoscope 118 is in the correct—or approximately correct—location withinthe patient. The “endo view” provides information about anatomicalstructures, e.g., the shape of an intestine or colon of the patient,around the distal end of the endoscope 118. The display modules 202 cansimultaneously display the 3D model and computerized tomography (CT)scans of the anatomy the around distal end of the endoscope 118.Further, the display modules 202 may overlay the already determinednavigation paths of the endoscope 118 on the 3D model and scans/imagesgenerated based on preoperative model data (e.g., CT scans).

In some embodiments, a model of the endoscope 118 is displayed with the3D models to help indicate a status of a surgical procedure. Forexample, the CT scans identify a lesion in the anatomy where a biopsymay be necessary. During operation, the display modules 202 may show areference image captured by the endoscope 118 corresponding to thecurrent location of the endoscope 118. The display modules 202 mayautomatically display different views of the model of the endoscope 118depending on user settings and a particular surgical procedure. Forexample, the display modules 202 show an overhead fluoroscopic view ofthe endoscope 118 during a navigation step as the endoscope 118approaches an operative region of a patient.

III. Instrument Device Manipulator

FIG. 3A shows an isometric view of an example independent drivemechanism of the IDM 117 shown in FIG. 1 , according to one embodiment.The independent drive mechanism can tighten or loosen the pull wires321, 322, 323, and 324 (e.g., independently from each other) of anendoscope by rotating the output shafts 305, 306, 307, and 308 of theIDM 117, respectively. Just as the output shafts 305, 306, 307, and 308transfer force down pull wires 321, 322, 323, and 324, respectively,through angular motion, the pull wires 321, 322, 323, and 324 transferforce back to the output shafts. The IDM 117 and/or the surgical roboticsystem 100 can measure the transferred force using a sensor, e.g., astrain gauge further described below.

FIG. 3B shows a conceptual diagram that shows how forces may be measuredby a strain gauge 334 of the independent drive mechanism shown in FIG.3A, according to one embodiment. A force 331 may direct away from theoutput shaft 305 coupled to the motor mount 333 of the motor 337.Accordingly, the force 331 results in horizontal displacement of themotor mount 333. Further, the strain gauge 334 horizontally coupled tothe motor mount 333 experiences strain in the direction of the force331. The strain may be measured as a ratio of the horizontaldisplacement of the tip 335 of strain gauge 334 to the overallhorizontal width 336 of the strain gauge 334.

In some embodiments, the IDM 117 includes additional sensors, e.g.,inclinometers or accelerometers, to determine an orientation of the IDM117. Based on measurements from the additional sensors and/or the straingauge 334, the surgical robotic system 100 can calibrate readings fromthe strain gauge 334 to account for gravitational load effects. Forexample, if the IDM 117 is oriented on a horizontal side of the IDM 117,the weight of certain components of the IDM 117 may cause a strain onthe motor mount 333. Accordingly, without accounting for gravitationalload effects, the strain gauge 334 may measure strain that did notresult from strain on the output shafts.

IV. Endoscope IV. A. Endoscope Top View

FIG. 4A shows a top view of an example endoscope 118, according to oneembodiment. The endoscope 118 includes a leader 415 tubular componentnested or partially nested inside and longitudinally-aligned with asheath 411 tubular component. The sheath 411 includes a proximal sheathsection 412 and distal sheath section 413. The leader 415 has a smallerouter diameter than the sheath 411 and includes a proximal leadersection 416 and distal leader section 417. The sheath base 414 and theleader base 418 actuate the distal sheath section 413 and the distalleader section 417, respectively, for example, based on control signalsfrom a user of a surgical robotic system 100. The sheath base 414 andthe leader base 418 are, e.g., part of the IDM 117 shown in FIG. 1 .

Both the sheath base 414 and the leader base 418 include drivemechanisms (e.g., the independent drive mechanism further described withreference to FIG. 3A-B in Section III. Instrument Device Manipulator) tocontrol pull wires coupled to the sheath 411 and leader 415. Forexample, the sheath base 414 generates tensile loads on pull wirescoupled to the sheath 411 to deflect the distal sheath section 413.Similarly, the leader base 418 generates tensile loads on pull wirescoupled to the leader 415 to deflect the distal leader section 417. Boththe sheath base 414 and leader base 418 may also include couplings forthe routing of pneumatic pressure, electrical power, electrical signals,or optical signals from IDMs to the sheath 411 and leader 415,respectively. A pull wire may include a steel coil pipe along the lengthof the pull wire within the sheath 411 or the leader 415, whichtransfers axial compression back to the origin of the load, e.g., thesheath base 414 or the leader base 418, respectively.

The endoscope 118 can navigate the anatomy of a patient with ease due tothe multiple degrees of freedom provided by pull wires coupled to thesheath 411 and the leader 415. For example, four or more pull wires maybe used in either the sheath 411 and/or the leader 415, providing eightor more degrees of freedom. In other embodiments, up to three pull wiresmay be used, providing up to six degrees of freedom. The sheath 411 andleader 415 may be rotated up to 360 degrees along a longitudinal axis406, providing more degrees of motion. The combination of rotationalangles and multiple degrees of freedom provides a user of the surgicalrobotic system 100 with a user friendly and instinctive control of theendoscope 118. Although not illustrated in FIG. 4A, the endoscope 118may include one or more optical fibers for sensing the shape in one ormore portions of the endoscope 118. For example, as illustrated in FIG.4B, the optical fiber(s) can be included in the leader portion of theendoscope 118. Alternatively or additionally, the optical fiber(s) canbe included in the sheath portion of the endoscope 118. As will beexplained in more detail below, information from the optical fibers canbe used in combination with information from other input sources, suchas other input sensors, modelling data, known properties andcharacteristics of the endoscope, and the like, to enhance performanceof the navigation system, catheter control, or the like.

IV. B. Endoscope Cross-Sectional View

FIG. 4B illustrates an example endoscope cross-section 430 of theendoscope 118 shown in FIG. 4A, according to one embodiment. In FIG. 4B,the endoscope cross-section 430 includes illumination sources 432,electromagnetic (EM) coils 434, and shape-sensing fibers 436. Theillumination sources 432 provide light to illuminate an interior portionof an anatomical space. The provided light may allow an imaging deviceprovided at the tip of the endoscope 118 to record images of that space,which can then be transmitted to a computer system such as commandconsole 200 for processing as described herein. EM coils 434 may be usedwith an EM tracking system to detect the position and orientation of thetip of the endoscope 118 while it is disposed within an anatomicalsystem. In some embodiments, the coils may be angled to providesensitivity to EM fields along different axes, giving the ability tomeasure a full 6 degrees of freedom: three positional and three angular.In other embodiments, only a single coil may be disposed within the tipof the endoscope 118, with its axis oriented along the endoscope shaftof the endoscope 118; due to the rotational symmetry of such a system,it is insensitive to roll about its axis, so only 5 degrees of freedommay be detected in such a case. The endoscope cross-section 430 furtherincludes a working channel 438 through which surgical instruments, suchas biopsy needles, may be inserted along the endoscope shaft, allowingaccess to the area near the endoscope tip. “Instruments” as used hereincan refer to surgical instruments, medical instruments, and any otherinstrument or device that can be navigated in a luminal network.

While the illustrated embodiment is disclosed as including illuminationsources 432 and EM coils 434 and corresponding imaging device and EMtracking system, it is anticipated that modified embodiments of theendoscopes described herein can be without one or more of such features.Further, while the shape-sensing fibers 436 are described as beingintegrated into the endoscope 118, in other embodiments, any of the oneor more shape-sensing fibers 436 can instead be a removable workingchannel device that can be inserted into the working channel 438 andremoved from the working channel 438 after shape sensing is performed.In other embodiments, the shape-sensing fibers 436 may be mountedexternal to the endoscope 118.

IV. C. Shape-Sensing Optical Fibers

FIG. 4C shows a system 450 having a shape detector 452, which can beused to generate and detect light used for determining the shape of theinstrument, an endoscope 454, and an optical fiber 456. The opticalfiber 456 can include one or more segments of fiber Bragg gratings (FBG)458, which reflect certain wavelengths of light while transmitting otherwavelengths. The gratings 458 may comprise a series of modulations ofrefractive index so as to generate a spatial periodicity in therefraction index. During fabrication of the gratings 458, themodulations can be spaced by a known distance, thereby causingreflection of a known band of wavelengths. The shape detector 452 maytransmit light through the optical fiber 456 and receive light reflectedfrom the optical fiber 456. The shape detector 452 may further generatereflection spectrum data based on the wavelengths of light reflected bythe gratings 458.

As shown in FIG. 4C, a single optical fiber may include multiple sets offiber Bragg gratings. The endoscope 454 may include multiple opticalfibers, and the shape detector 452 may detect and analyze signals frommore than one fiber. One or more optical fibers may be included in theleader 415 of FIG. 4A, the sheath 411 of FIG. 4A, or both. Although theendoscope 454 is used as an example, the techniques described herein canbe applied to any other elongated instrument. The shape detector 452 maybe operatively coupled with a controller configured to determine ageometric shape or configuration of the optical fiber 456 and,therefore, at least a portion of the endoscope 454 (or other elongatedinstrument such as a catheter and the like) based on a spectral analysisof the detected reflected light signals.

The controller within or in communication with the shape detector 452(e.g., the surgical robotic system 500) can analyze the reflectionspectrum data to generate position and orientation data of the endoscope454 in two or three dimensional space. In particular, as the endoscope454 bends, the optical fiber 456 positioned inside also bends, whichcauses strain on the optical fiber 456. When strain is induced on theoptical fiber 456, the spacing of the modulations will change, dependingon the amount of strain on the optical fiber 456. To measure strain,light is sent down the optical fiber 456, and characteristics of thereturning light are measured. For example, the gratings 458 may producea reflected wavelength that is a function of the strain on the opticalfiber 456 (and other factors such as temperature). Based on the specificwavelengths of light reflected by the gratings 458, the system candetermine the amount of strain on the optical fiber 456 and furtherpredict the shape of the optical fiber 456 based on the amount of strain(e.g., based on how the strain characteristics of a “straight” endoscopemay differ from those of a “curved” endoscope). Thus, the system candetermine, for example, how many degrees the endoscope 454 has bent inone or more directions (e.g., in response to commands from the surgicalrobotic system 500) by identifying differences in the reflectionspectrum data.

In some embodiments, the optical fiber 456 includes multiple coreswithin a single cladding. In such embodiments, each core may operate asa separate optical fiber with sufficient distance and claddingseparating the cores such that the light in each core does not interactsignificantly with the light carried in other cores. In otherembodiments, the number of cores may vary or each core may be containedin a separate optical fiber. When the strain and shape analysis isapplied to a multicore optical fiber, bending of the optical fiber 456may induce strain on the cores that can be measured by monitoring thewavelength shifts in each core. By having two or more cores disposedoff-axis in the optical fiber 456, bending of the optical fiber inducesdifferent strains on each of the cores. These strains are a function ofthe local degree of bending of the fiber. For example, regions of thecores containing the gratings 458, if located at points where theoptical fiber 456 is bent, can thereby be used to determine the amountof bending at those points. These data, combined with the known spacingsof the gratings 458, can be used to reconstruct the shape of the opticalfiber 456.

The optical fiber is suitable for data collection inside the body of thepatient because no line-of-sight to the shape sensing optical fiber isrequired. Various systems and methods for monitoring the shape andrelative position of an optical fiber in three dimensions are describedin U.S. Patent Application Publication No. 2006/0013523, filed Jul. 13,2005, titled “FIBER OPTIC POSITION AND SHAPE SENSING DEVICE AND METHODRELATING THERETO,” and U.S. Pat. No. 6,389,187, filed on Jun. 17, 1998,entitled “OPTICAL FIBER BEND SENSOR,” the contents of which are fullyincorporated herein by reference.

While the illustrated embodiment utilizes a fiber with Bragg gratings,in a modified variation, an optical fiber can include slightimperfections that result in index of refraction variations along thefiber core. These variations can result in a small amount of backscatterthat is called Rayleigh scatter. Changes in strain or temperature of theoptical fiber cause changes to the effective length of the opticalfiber. This change in the effective length results in variation orchange of the spatial position of the Rayleigh scatter points. Crosscorrelation techniques can measure this change in the Rayleighscattering and can extract information regarding the strain. Thesetechniques can include using optical frequency domain reflectometertechniques in a manner that is very similar to that associated with lowreflectivity fiber gratings.

Methods and devices for calculating birefringence in an optical fiberbased on Rayleigh scatter as well as apparatus and methods for measuringstrain in an optical fiber using the spectral shift of Rayleigh scattercan be found in PCT Publication No. WO 2006/099056 filed on Mar. 9, 2006and U.S. Pat. No. 6,545,760 filed on Mar. 24, 2000, both of which arefully incorporated herein by reference. Birefringence can be used tomeasure axial strain and/or temperature in a waveguide.

IV. D. Improving Strain-Based Shape Data

Strain-based shape sensing can allow reconstruction of the shape of anendoscope or other instrument by measuring the strain along the opticalfibers that run inside the instrument. The measurement of the straincaptures the spatiotemporal variations of the reflection of light ongratings inside the optical fibers. The distance between each gratingaffects the reflection and can therefore be used to measure the strainat a precise location along the optical fiber (or the instrument).However, in some cases, strain-based shape sensing can be negativelyaffected by noise. In such cases, it can be difficult to distinguishbetween a real change in strain and a false one.

An improved strain-based shape sensing system can utilize other dataavailable to the system (e.g., robotic data, image data, EM data, etc.)to improve the precision of (or adjust the confidence in) itsstrain-based shape sensing or state estimations determined based on suchstrain-based shape sensing. Alternatively or additionally, an improvedstrain-based shape sensing system can utilize the shape data determinedbased on its strain-based shape sensing to improve the precision of (oradjust the confidence in) its other data (e.g., robotic data, imagedata, EM data, etc.) or state estimations determined based on such data.

FIGS. 4D-4E illustrate how the system may utilize information availableto the system to improve, adjust, or weight its strain-based shapesensing. The system may access data available to the system such asrobotic data (e.g., command data, force and distance data, mechanicalmodel data, kinematic model data, etc.) and determine, based on suchdata, certain characteristics about the shape of the instrument (orspecific portions thereof) navigated within a patient's body. Suchcharacteristics may include curvature information (e.g., maximumcurvature that the instrument is capable of exhibiting, or a range ofacceptable curvature values given the current force and distance dataindicated by the robotic data), movement information (e.g., maximumspeed at which the instrument is capable of moving, or a range ofacceptable speed values given the current force and distance dataindicated by the robotic data), sheath information (e.g., current shapeof the sheath covering one or more portions of the instrument), and thelike. Upon determining that the strain-based shape prediction does notsatisfy one or more constraints indicated by these characteristicsdetermined based on the robotic data, the system can adjust thestrain-based shape prediction such that the adjusted strain-based shapeprediction satisfies the constraints, reduce the confidence or weightassociated with the particular strain-based shape prediction, ordisregard the strain-based shape prediction.

FIG. 4D shows actual shape 472 of the endoscope 118, robotic-data-basedshape prediction 473, and strain-based shape prediction 474 of theendoscope 118. The actual shape 472 exhibits an actual curvature 476,whereas the robotic-data-based shape prediction 473 exhibit a predictedcurvature 477, and the strain-based shape prediction 474 exhibits apredicted curvature 478. In the example of FIG. 4D, the system maydetermine, based on the robotic data, one or more conditions that theendoscope 118 is expected to satisfy (e.g., the curvature value at agiven point along the endoscope 118 should be within a predeterminedrange of values, or should be within a range of values determined basedon the pull force on the pull wires and/or the distances that the pullwires have been actuated). Upon determining that a portion of thestrain-based shape prediction 474 does not satisfy such conditions(e.g., by indicating a predicted curvature value that is outside anexpected curvature value range as determined based on the robotic datacorresponding to the endoscope 118), the system can adjust thestrain-based shape prediction 474 such that the portion of thestrain-based shape prediction 474 satisfies the conditions (e.g., suchthat the shape data no longer indicates a predicted curvature value thatis outside the expected curvature value range), reduce the confidence orweight associated with the portion of the strain-based shape prediction474, or disregard the portion of the strain-based shape prediction 474(e.g., refrain from using the portion of the strain-based shapeprediction 474 in estimating the current state of the endoscope 118).For example, as shown in FIG. 4D, the system may determine, based on therobotic data (e.g., pull force and distances), the robotic-data-basedshape prediction 473 exhibiting the predicted curvature 477 at a givenpoint. The system may compare the predicted curvature 477 to thepredicted curvature 478 exhibited by the strain-based shape prediction474. Upon determining that the predicted curvature 478 is different fromthe predicted curvature 477, the system may adjust the predictedcurvature 487 to equal the predicted curvature 477. Alternatively, upondetermining that the predicted curvature 478 is not within a giventhreshold range (e.g., ±10 percent) from the predicted curvature 477,the system may adjust the predicted curvature 487 to be within thethreshold range (e.g., set to the upper bound of the threshold range ifthe predicted curvature 478 exceeds the range, and set to the lowerbound of the threshold range if the predicted curvature 478 falls shortof the range). Additionally or alternatively, the system may lower theconfidence value associated with the strain-based shape prediction 474based on a determination that the predicted curvature 478 is differentfrom the predicted curvature 477 (or that the predicted curvature 478 isnot within the given threshold range), and/or increase the confidencevalue associated with the strain-based shape prediction 474 based on adetermination that the predicted curvature 478 equals the predictedcurvature 477 (or that the predicted curvature 478 is within the giventhreshold range).

FIG. 4E shows robotic-data-based shape prediction 482 of the endoscope118 and strain-based shape prediction 484 of the endoscope 118. Therobotic-data-based shape prediction 482 exhibits a predicted movement486, whereas the strain-based shape prediction 484 exhibits a predictedmovement 488. As described with reference to FIG. 4D, the system maydetermine, based on the robotic data, one or more conditions that theendoscope 118 is expected to satisfy. For example, based on the roboticdata, the system may determine that the speed at which the endoscope 118moves should be within a certain range of values). Upon determining thata portion of the strain-based shape prediction 484 does not satisfy suchconditions, the system can adjust the strain-based shape prediction 484such that the portion of the strain-based shape prediction 484 satisfiesthe conditions (e.g., such that the shape data no longer indicates apredicted speed value that is outside the expected speed value range),reduce the confidence or weight associated with the portion of thestrain-based shape prediction 484, or disregard the portion of thestrain-based shape prediction 484 (e.g., refrain from using the portionof the strain-based shape prediction 484 in estimating the current stateof the endoscope 118). For example, as shown in FIG. 4E, the system maydetermine, based on the robotic data (e.g., pull force and/or distancesas a function of time), the robotic-data-based shape prediction 482exhibiting the predicted movement 486. The system may compare thepredicted movement 486 to the predicted movement 488 exhibited by thestrain-based shape prediction 484. Upon determining that the predictedmovement 488 is different from the predicted movement 486, the systemmay adjust the predicted movement 488 to be identical to the predictedmovement 486. Alternatively, upon determining that the predictedmovement 488 is not within a given threshold range (e.g., movement speedbeing within ±10 percent of the movement speed of the predicted movement486), the system may adjust the predicted movement 488 to be within thethreshold range (e.g., set to the upper bound of the threshold movementspeed range if the predicted movement speed of the strain-based shapeprediction 484 exceeds the movement speed range, and set to the lowerbound of the threshold movement speed range if the predicted movementspeed of the strain-based shape prediction 484 falls short of themovement speed range). Additionally or alternatively, the system maylower the confidence value associated with the strain-based shapeprediction 484 based on a determination that the predicted movement 488is different from the predicted movement 486 (or that the predictedmovement 488 is not within the given threshold), and/or increase theconfidence value associated with the strain-based shape prediction 484based on a determination that the predicted movement 488 is identical tothe predicted movement 486 (or that the predicted movement 488 is withinthe given threshold).

The process of collecting strain data and other data (some or all ofwhich can be utilized to improve the strain-based shape data) anddetermining state estimations is described in greater detail below withreference to FIGS. 8-11 .

V. Registration Transform of EM System to 3D Model V. A. Schematic Setupof an EM Tracking System

In certain embodiments an EM tracking system can be used in combinationwith the systems described herein. FIG. 5 shows an example schematicsetup of such an EM tracking system 505 that can be included in asurgical robotic system 500, according to one embodiment. In FIG. 5 ,multiple robot components (e.g., window field generator, referencesensors as described below) are included in the EM tracking system 505.The surgical robotic system 500 includes a surgical bed 511 to hold apatient's body. Beneath the bed 511 is the window field generator (WFG)512 configured to sequentially activate a set of EM coils (e.g., the EMcoils 434 shown in FIG. 4B). The WFG 512 generates an alternatingcurrent (AC) magnetic field over a wide volume; for example, in somecases it may create an AC field in a volume of about 0.5×0.5×0.5 m.

Additional fields may be applied by further field generators to aid intracking instruments within the body. For example, a planar fieldgenerator (PFG) may be attached to a system arm adjacent to the patientand oriented to provide an EM field at an angle. Reference sensors 513may be placed on the patient's body to provide local EM fields tofurther increase tracking accuracy. Each of the reference sensors 513may be attached by cables 514 to a command module 515. The cables 514are connected to the command module 515 through interface units 516which handle communications with their respective devices as well asproviding power. The interface unit 516 is coupled to a system controlunit (SCU) 517 which acts as an overall interface controller for thevarious entities mentioned above. The SCU 517 also drives the fieldgenerators (e.g., WFG 512), as well as collecting sensor data from theinterface units 516, from which it calculates the position andorientation of sensors within the body. The SCU 517 may be coupled to apersonal computer (PC) 518 to allow user access and control.

The command module 515 is also connected to the various IDMs 519 coupledto the surgical robotic system 500 as described herein. The IDMs 519 aretypically coupled to a single surgical robotic system (e.g., thesurgical robotic system 500) and are used to control and receive datafrom their respective connected robotic components; for example, roboticendoscope tools or robotic arms. As described above, as an example, theIDMs 519 are coupled to an endoscopic tool (not shown here) of thesurgical robotic system 500.

The command module 515 receives data passed from the endoscopic tool.The type of received data depends on the corresponding type ofinstrument attached. For example, example received data includes sensordata (e.g., image data, EM data), robotic data (e.g., command data,force and distance data, mechanical model data, kinematic model data,etc.), control data, and/or video data. To better handle video data, afield-programmable gate array (FPGA) 520 may be configured to handleimage processing. Comparing data obtained from the various sensors,devices, and field generators allows the SCU 517 to precisely track themovements of different components of the surgical robotic system 500,and for example, positions and orientations of these components.

In order to track a sensor through the patient's anatomy, the EMtracking system 505 may require a process known as “registration,” wherethe system finds the geometric transformation that aligns a singleobject between different coordinate systems. For instance, a specificanatomical site on a patient has two different representations in the 3Dmodel coordinates and in the EM sensor coordinates. To be able toestablish consistency and common language between these two differentcoordinate systems, the EM tracking system 505 needs to find thetransformation that links these two representations, i.e., registration.For example, the position of the EM tracker relative to the position ofthe EM field generator may be mapped to a 3D coordinate system toisolate a location in a corresponding 3D model.

V. B. 3D Model Representation

FIGS. 6A-6B show an example anatomical lumen 600 and an example 3D model620 of the anatomical lumen, according to one embodiment. Morespecifically, FIGS. 6A-6B illustrate the relationships of centerlinecoordinates, diameter measurements and anatomical spaces between theactual anatomical lumen 600 and its 3D model 620. In FIG. 6A, theanatomical lumen 600 is roughly tracked longitudinally by centerlinecoordinates 601, 602, 603, 604, 605, and 606 where each centerlinecoordinate roughly approximates the center of the tomographic slice ofthe lumen. The centerline coordinates are connected and visualized by acenterline 607. The volume of the lumen can be further visualized bymeasuring the diameter of the lumen at each centerline coordinate, e.g.,diameters 608, 609, 610, 611, 612, and 613 represent the measurements ofthe anatomical lumen 600 corresponding to coordinates 601, 602, 603,604, 605, and 606.

FIG. 6B shows the example 3D model 620 of the anatomical lumen 600 shownin FIG. 6A, according to one embodiment. In FIG. 6B, the anatomicallumen 600 is visualized in 3D space by first locating the centerlinecoordinates 601, 602, 603, 604, 605, and 606 in 3D space based on thecenterline 607. As one example, at each centerline coordinate, the lumendiameter is visualized as a 2D circular space (e.g., the 2D circularspace 630) with diameters 608, 609, 610, 611, 612, and 613. Byconnecting those 2D circular spaces to form a 3D space, the anatomicallumen 600 is approximated and visualized as the 3D model 620. Moreaccurate approximations may be determined by increasing the resolutionof the centerline coordinates and measurements, i.e., increasing thedensity of centerline coordinates and measurements for a given lumen orsubsection. Centerline coordinates may also include markers to indicatepoint of interest for the physician, including lesions.

In some embodiments, a pre-operative software package is also used toanalyze and derive a navigation path based on the generated 3D model ofthe anatomical space. For example, the software package may derive ashortest navigation path to a single lesion (marked by a centerlinecoordinate) or to several lesions. This navigation path may be presentedto the operator intra-operatively either in two-dimensions orthree-dimensions depending on the operator's preference. In certainimplementations, the navigation path (or at a portion thereof) may bepre-operatively selected by the operator. The path selection may includeidentification of one or more target locations (also simply referred toas a “target”) within the patient's anatomy.

FIG. 7 shows a computer-generated 3D model 700 representing ananatomical space, according to one embodiment. As discussed above inFIGS. 6A-6B, the 3D model 700 may be generated using a centerline 701that was obtained by reviewing CT scans that were generatedpreoperatively. In some embodiments, computer software may be able tomap a navigation path 702 within the tubular network to access anoperative site 703 (or other target) within the 3D model 700. In someembodiments, the operative site 703 may be linked to an individualcenterline coordinate 704, which allows a computer algorithm totopologically search the centerline coordinates of the 3D model 700 forthe optimum path 702 within the tubular network. In certain embodiments,the topological search for the path 702 may be constrained by certainoperator selected parameters, such as the location of one or moretargets, one or more waypoints, etc.

In some embodiments, the distal end of the endoscopic tool within thepatient's anatomy is tracked, and the tracked location of the endoscopictool within the patient's anatomy is mapped and placed within a computermodel, which enhances the navigational capabilities of the tubularnetwork. In order to track the distal working end of the endoscopictool, i.e., location and orientation of the working end, a number ofapproaches may be employed, either individually or in combination.

In a sensor-based approach to localization, a sensor, such as an EMtracker, may be coupled to the distal working end of the endoscopic toolto provide a real-time indication of the progression of the endoscopictool. In EM-based tracking, an EM tracker, embedded in the endoscopictool, measures the variation in the electromagnetic field created by oneor more EM transmitters. The transmitters (or field generators), may beplaced close to the patient (e.g., as part of the surgical bed) tocreate a low intensity magnetic field. This induces small-currents insensor coils in the EM tracker, which are correlated to the distance andangle between the sensor and the generator. The electrical signal maythen be digitized by an interface unit (on-chip or PCB) and sent viacables/wiring back to the system cart and then to the command module.The data may then be processed to interpret the current data andcalculate the precise location and orientation of the sensor relative tothe transmitters. Multiple sensors may be used at different locations inthe endoscopic tool, for instance in leader and sheath in order tocalculate the individual positions of those components. Accordingly,based on readings from an artificially-generated EM field, the EMtracker may detect changes in field strength as it moves through thepatient's anatomy.

VI. Navigation Configuration System VI. A. High-Level Overview ofNavigation Configuration System

FIG. 8A shows an example block diagram of a navigation configurationsystem 900, according to one embodiment. In FIG. 8A, the navigationconfiguration system 900 includes multiple input data stores, anavigation module 905 that receives various types of input data from themultiple input data stores, and an output navigation data store 990 thatreceives output navigation data from the navigation module 905. Theblock diagram of the navigation configuration system 900 shown in FIG.8A is merely one example, and in alternative embodiments not shown, thenavigation configuration system 900 can include different and/oradditional elements or not include one or more of the elements shown inFIG. 8A. Likewise, functions performed by various elements of thenavigation configuration system 900 may differ according to differentembodiments. The navigation configuration system 900 may be similar tothe navigational system described in U.S. Pat. No. 9,727,963, issued onAug. 8, 2017, the entirety of which is incorporated herein by reference.

The input data, as used herein, refers to raw or processed data gatheredfrom input devices (e.g., command module, optical sensor, EM sensor,IDM) for generating estimated state information for the endoscope 118(or other instrument) as well as output navigation data. The multipleinput data stores 901-941 can include a strain data store 901, an imagedata store 910, an EM data store 920, a robotic data store 930, a 3Dmodel data store 940, and other data store(s) 941. Each type of theinput data stores 901-941 stores the name-indicated type of data foraccess and use by the navigation module 905. Strain data may include oneor more measurements of strain along the endoscope 118 (e.g., generatedand/or stored by the shape detector 452 of FIG. 4C).

Image data may include one or more image frames captured by the imagingdevice at the instrument tip, as well as information such as frame ratesor timestamps that allow a determination of the time elapsed betweenpairs of frames.

Robotic data may include data typically used by the system for functionsrelated to the control of the instrument (e.g., endoscope 118 and/or itssheath), and/or physical movement of the instrument or part of theinstrument (e.g., the instrument tip or sheath) within the tubularnetwork. Robotic data may allow the state of the instrument to beinferred based on the data measured while navigating the instrumentwithin the tubular network. The kinematic and dynamic models may begenerated based on a priori information gathered during a calibrationstage. This a priori information can be stored on the device and readand utilized by the robot to improve the drive, control, and navigationof the instrument and to improve other types of data available to therobot (e.g., EM data, image data, strain-based shape data, etc.). Therobotic data may include parameters that are specific to eachinstrument.

FIG. 8C illustrates examples of the robotic data that may be stored inthe robotic data store 930 of FIG. 8A. As shown in FIG. 8C, the roboticdata may include command data 931 instructing the instrument tip toreach a specific anatomical site and/or change its orientation (e.g.,articulation data instructing the instrument to exhibit a desiredarticulation with a specific pitch, roll, and yaw, insertion andretraction data instructing the insertion and retraction for one or bothof a leader and a sheath, etc.) force and distance data 932 (e.g., thedistances that the pull wires have been actuated since the device wasloaded on the robot, the amount of force being applied to the pull wiresas measured by the torque sensors in the IDM, the amount of insertionforce exerted by the robot arm to insert or retract the instruments,etc.), mechanical model data 933 representing mechanical movement of anelongate member of the instrument (e.g. motion of one or more pullwires, tendons, or shafts of the endoscope that drive the actualmovement of the medical instrument within the tubular network, kinematicmodel data 934 representing the motion and shape of an instrument (e.g.,geometric parameters indicative of a position of the instrument, and/orany changes to the geometric parameters relative to a pre-determined orreference position or set of coordinates), and the like.

EM data may be collected by one or more EM sensors (e.g., locatedproximal to a tip of the instrument) and/or the EM tracking system asdescribed above. 3D model data may be derived from, among other things,2D CT scans as described above.

The output navigation data store 990 receives and stores outputnavigation data provided by the navigation module 905. Output navigationdata indicates information to assist in directing the instrument througha patient's anatomy and in one example through a tubular network toarrive at a particular destination within the tubular network, and isbased on estimated state information for the instrument at each instanttime. The estimated state information can include the location andorientation of the instrument within the tubular network. In oneembodiment, as the instrument moves inside the tubular network, theoutput navigation data indicating updates of movement andlocation/orientation information of the instrument is provided in realtime, which better assists its navigation through the tubular network.

To determine the output navigation data, the navigation module 905locates (or determines) the estimated state of the instrument within thetubular network. As shown in FIG. 8A, the navigation module 905 furtherincludes various algorithm modules, such as a strain-based algorithmmodule 945, an EM-based algorithm module 950, an image-based algorithmmodule 960, a robot-based algorithm module 970, an algorithm module 971based on other data, etc. These modules may each consume mainly certaintypes of input data and contribute a different type of data to a stateestimator 980. As illustrated in FIG. 8A, the different kinds of dataoutput by these modules, (labeled strain-based estimated state data,EM-based estimated state data, the image-based estimated state data, andthe robot-based estimated state data, and estimated state data based onother data) may be generally referred to as “intermediate data” for thesake of explanation. In some cases, the navigation module 905determines, based on the estimated state of the instrument, that adamage to the instrument or a malfunction is imminent (e.g., buckling,prolapse, etc.). In such cases, the navigation module 905 may cause theinstrument to be controlled in a way to avoid the damage or malfunction.The detailed composition of each algorithm module and the stateestimator 980 is described in greater detail below.

VI. B. Navigation Module VI. B. 1. State Estimator

As introduced above, the navigation module 905 further includes a stateestimator 980 as well as multiple algorithm modules that employdifferent algorithms for navigating through a tubular network. Forclarity of description, the state estimator 980 is described first,followed by the description of the various modules that exchange datawith the state estimator 980.

The state estimator 980 included in the navigation module 905 receivesvarious intermediate data and provides the estimated state of theinstrument tip (or other portions of the instrument) as a function oftime, where the estimated state indicates the estimated location andorientation information of the instrument tip (or other portions of theinstrument) within the tubular network. The estimated state data arestored in the estimated state data store 985 that is included in thestate estimator 980. While the description herein is described withinthe context of determining the estimated location and orientationinformation of the instrument tip (or other portions of the instrument)within a tubular network, in other arrangements, the information can beused to determine estimated location and orientation information of theinstrument tip (or other portions of the instrument) with respect to thepatient, in general.

VI. B. 2. Estimated State Data Store

The estimated state data store 985 may include a bifurcation data store,a position data store, a depth data store, and an orientation datastore. However this particular breakdown of data storage is merely oneexample, and in alternative embodiments not shown, different and/oradditional data stores can be included in the estimated state data store985.

The various stores introduced above represent estimated state data in avariety of ways. Bifurcation data may refer to the location of theinstrument with respect to the set of branches (e.g., bifurcation,trifurcation or a division into more than three branches) within thetubular network. For example, the bifurcation data can be set of branchchoices elected by the instrument as it traverses through the tubularnetwork, based on a larger set of available branches as provided, forexample, by the 3D model which maps the entirety of the tubular network.The bifurcation data can further include information in front of thelocation of the instrument tip, such as branches (bifurcations) that theinstrument tip is near but has not yet traversed through, but which mayhave been detected, for example, based on the tip's current positioninformation relative to the 3D model, or based on images captured of theupcoming bifurcations.

Position data may indicate three-dimensional position of some part ofthe instrument within the tubular network or some part of the tubularnetwork itself. Position data can be in the form of absolute locationsor relative locations relative to, for example, the 3D model of thetubular network. As one example, position data can include an indicationof the position of the location of the instrument being within aspecific branch. The identification of the specific branch may also bestored as a segment identification (ID) which uniquely identifies thespecific segment of the model in which the instrument tip is located.

Depth data may indicate depth information of the instrument tip withinthe tubular network. Example depth data includes the total insertion(absolute) depth of the instrument into the patient as well as the(relative) depth within an identified branch (e.g., the segmentidentified by the position data store 1087). Depth data may bedetermined based on position data regarding both the tubular network andinstrument.

Orientation data may indicate orientation information of the instrumenttip, and may include overall roll, pitch, and yaw in relation to the 3Dmodel as well as pitch, roll, yaw within an identified branch.

VI. B. 3. Data Output to Algorithm Modules

As illustrated in FIG. 8A, the state estimator 980 provides theestimated state data back to the algorithm modules for generating moreaccurate intermediate data, which the state estimator uses to generateimproved and/or updated estimated states, and so on forming a feedbackloop. The state estimator 980 receives estimated state data from one ormore algorithm modules shown in FIG. 8A. The state estimator 980 usesthis data to generate “estimated state data (prior)” that is associatedwith timestamp “t−1.” The state estimator 980 then provides the data toone or more (which can be a different combination of algorithm modulesthan the one from which estimated state data was received previously) ofthe algorithm modules. The “estimated state data (prior)” may be basedon a combination of different types of intermediate data (e.g., imagedata, mechanical model data, command data, kinematic model data, and thelike) that is associated with timestamp “t−1” as generated and receivedfrom different algorithm modules. For example, estimated state databased on a combination of the non-strain-based data 972 may be providedto the strain-based algorithm module 945, and the strain-based algorithmmodule 945 may determine and output strain-based estimated state data tothe state estimator 980.

Next, the one or more of the algorithm modules run their respectivealgorithms using the received estimated state data (prior) to output tothe state estimator 980 improved and updated estimated state data, whichis represented by “estimated state data (current)” shown for therespective algorithm modules and associated with timestamp “t.” Thisprocess can be repeated for future timestamps to generate estimatedstate data.

As the state estimator 980 may use several different kinds ofintermediate data to arrive at its estimates of the state of theinstrument within the tubular network, the state estimator 980 isconfigured to account for the various different kinds of errors anduncertainty in both measurement and analysis that each type ofunderlying data (robotic, EM, image) and each type of algorithm modulemay create or carry through into the intermediate data used forconsideration in determining the estimated state. To address these, twoconcepts are discussed, that of a probability distribution and that ofconfidence value.

The term “probability” in the phrase “probability distribution”, as usedherein, refers to a likelihood of an estimation of a possible locationand/or orientation of the instrument being correct. For example,different probabilities may be calculated by one of the algorithmmodules indicating the relative likelihood that the instrument is in oneof several different possible branches within the tubular network. Inone embodiment, the type of probability distribution (e.g., discretedistribution or continuous distribution) is chosen to match features ofan estimated state (e.g., type of the estimated state, for examplecontinuous position information vs. discrete branch choice). As oneexample, estimated states for identifying which segment the instrumentis in for a trifurcation may be represented by a discrete probabilitydistribution, and may include three discrete values of 20%, 30% and 50%representing chance as being in the location inside each of the threebranches as determined by one of the algorithm modules. As anotherexample, the estimated state may include a roll angle of the instrumentof 40±5 degrees and a segment depth of the instrument tip within abranch may be is 4±1 mm, each represented by a Gaussian distributionwhich is a type of continuous probability distribution. Differentmethods or modalities can be used to generate the probabilities, whichwill vary by algorithm module as more fully described below withreference to later figures.

In contrast, the “confidence value,” as used herein, reflects a measureof confidence in the estimation of the state provided by one of thealgorithms based one or more factors. For strain-based algorithms usingshape-sensing fibers, factors such as temperature, proximity to theproximal end of the catheter, and the like may affect the confidence inestimation of the state. For example, thermal expansion and contractionof the optical fiber portions may erroneously indicate that theinstrument is bending. Further, in some embodiments, strain measurementsof distal portions of the instrument rely on shape/position datadetermined based on strain measurements of proximal portions of theinstrument (e.g., closer to the shape detector 452), and any errors inthe strain measurements of proximal portions may be magnified in thestrain measurements of distal portions. For the EM-based algorithms,factors such as distortion to EM field, inaccuracy in EM registration,shift or movement of the patient, and respiration of the patient mayaffect the confidence in estimation of the state. Particularly, theconfidence value in estimation of the state provided by the EM-basedalgorithms may depend on the respiration cycle of the patient, movementof the patient or the EM field generators, and the location within theanatomy where the instrument tip locates. For the image-basedalgorithms, examples factors that may affect the confidence value inestimation of the state include illumination condition for the locationwithin the anatomy where the images are captured, presence of fluid,tissue, or other obstructions against or in front of the optical sensorcapturing the images, respiration of the patient, condition of thetubular network of the patient itself (e.g., lung) such as the generalfluid inside the tubular network and occlusion of the tubular network,and specific operating techniques used in, e.g., navigating or imagecapturing.

For example one factor may be that a particular algorithm has differinglevels of accuracy at different depths in a patient's lungs, such thatrelatively close to the airway opening, a particular algorithm may havea high confidence in its estimations of instrument location andorientation, but the further into the bottom of the lung the instrumenttravels that confidence value may drop. Generally, the confidence valueis based on one or more systemic factors relating to the process bywhich a result is determined, whereas probability is a relative measurethat arises when trying to determine the correct result from multiplepossibilities with a single algorithm based on underlying data.

As one example, a mathematical equation for calculating results of anestimated state represented by a discrete probability distribution(e.g., branch/segment identification for a trifurcation with threevalues of an estimated state involved) can be as follows:

S ₁ =C _(EM) *P _(1,EM) +C _(Image) *P _(1,Image) +C _(Robot) *P_(1,Robot);

S ₂ =C _(EM) *P _(2,EM) +C _(Image) *P _(2,Image) +C _(Robot) *P_(2,Robot);

S ₃ =C _(EM) *P _(3,EM) +C _(Image) *P _(3,Image) +C _(Robot) *P_(3,Robot).

In the example mathematical equation above, S_(i) (i=1,2,3) representspossible example values of an estimated state in a case where 3 possiblesegments are identified or present in the 3D model, C_(EM), C_(Image),and C_(Robot) represents confidence value corresponding to EM-basedalgorithm, image-based algorithm, and robot-based algorithm andP_(i,EM), P_(i,Image), and P_(i,Robot) represent the probabilities forsegment i.

To better illustrate the concepts of probability distributions andconfidence value associated with estimate states, a detailed example isprovided here. In this example, a user is trying to identify segmentwhere an instrument tip is located in a certain trifurcation within acentral airway (the predicted region) of the tubular network, and threealgorithms modules are used including EM-based algorithm, image-basedalgorithm, and robot-based algorithm. In this example, a probabilitydistribution corresponding to the EM-based algorithm may be 20% in thefirst branch, 30% in the second branch, and 50% in the third (last)branch, and the confidence value applied to this EM-based algorithm andthe central airway is 80%. For the same example, a probabilitydistribution corresponding to the image-based algorithm may be 40%, 20%,40% for the first, second, and third branch, and the confidence valueapplied to this image-based algorithm is 30%; while a probabilitydistribution corresponding to the robot-based algorithm may be 10%, 60%,30% for the first, second, and third branch, and the confidence valueapplied to this image-based algorithm is 20%. The difference ofconfidence values applied to the EM-based algorithm and the image-basedalgorithm indicates that the EM-based algorithm may be a better choicefor segment identification in the central airway, compared with theimage-based algorithm. An example mathematical calculation of a finalestimated state can be: for the first branch:20%*80%+40%*30%+10%*20%=30%; for the second branch:30%*80%+20%*30%+60%*20%=42%; and for the third branch:50%*80%+40%*30%+30%*20%=58%.

In this example, the output estimated state for the instrument tip canbe the result values (e.g., the resulting 30%, 42%, and 58%), orderivative value from these result values such as the determination thatthe instrument tip is in the third branch.

As above the estimated state may be represented in a number of differentways. For example, the estimated state may further include an absolutedepth from airway to location of the tip of the instrument, as well as aset of data representing the set of branches traversed by the instrumentwithin the tubular network, the set being a subset of the entire set ofbranches provided by the 3D model of the patient's lungs, for example.The application of probability distribution and confidence value onestimated states allows improved accuracy of estimation of locationand/or orientation of the instrument tip within the tubular network.

VI. B. 4. Strain-Based Algorithm Module

VI. B. 4. i. Elements of Strain-Based Algorithm Module

The strain-based algorithm module 945 uses strain data to determine theestimated state of the instrument within the tubular network. FIGS. 8Band 9-11 illustrate the modules that may be included in the strain-basedalgorithm module 945. As illustrated in FIG. 8B, the strain-basedalgorithm module 945 may include (i) shape data determination module 906for determining the shape data based on strain data, (ii) shape datacomparison module 907 for comparing the shape data to robotic data,(iii) shape data adjustment module 908 for adjusting the shape data (orthe confidence in the shape data) based on the comparison between theshape data and the robotic data, and (iv) shape-based state estimationmodule 909 for determining shape-based estimated state data based on theadjusted shape data (or adjusted confidence in the shape data). Althoughillustrated as separate components, the modules 906-909 may beimplemented as one or more hardware components (e.g., a singlecomponent, individual components, or any number of components), one ormore software components (e.g., a single component, individualcomponents, or any number of components), or any combination thereof.The modules 906-909 are described in greater detail below with referenceto FIGS. 9-11 .

VI. B. 4. ii. Determining Shape Data

FIG. 9 shows an example shape data determination module that may beincluded in the strain-based algorithm module 945. As shown in FIG. 9 ,the shape data determination module 906 receives strain data from thestrain data store 901 and outputs shape data to shape data store 902.The shape data determination module 906 may determine the shape databased on the strain data received from the strain data store 901. Asdiscussed with reference to FIG. 8A, the strain data may include one ormore measurements of strain along the one or more optical fibers 456 (orone or more cores therein) that are generated and/or stored by the shapedetector 452 of FIG. 4C. The shape data may include angles, coordinates,or a combination thereof indicative of the current shape of theinstrument. In some cases, the shape data may include curvatureinformation (e.g., curvature value of one or more portion of theinstrument), orientation information (e.g., roll, pitch, and/or yaw ofone or more portions of the instrument), position information (e.g.,position of one or more portions of the instrument in a referencecoordinate system, which is, for example, used by the system to navigatethe instrument), and/or other information that can be used to indicatethe shape of the instrument.

VI. B. 4. iii. Improving Shape Data Using Robotic Data

FIG. 10 shows an example shape data comparison module and shape dataadjustment module that may be included in the strain-based algorithmmodule 945. As shown in FIG. 10 , the shape data comparison module 907receives data from a plurality of data stores 902-941. For example, thereceived data can include shape data from the shape data store 902 androbotic data from the robotic data store 930. The shape data comparisonmodule 907 may compare the received shape data to the received roboticdata and determine whether the received shape data is consistent withthe received robotic data.

As described herein, the robotic data may include, as one example,kinematic model data that indicates the movement of the instrumentexpected to result from a given set of control commands. The shape datacomparison module 907 may compare the movement indicated by the roboticdata with the movement indicated by the shape data received from theshape data determination module 906. Based on the comparison, the shapedata comparison module 907 may output a comparison result that indicateswhether or not the shape data is consistent with the robotic data andthe extent of the difference between the shape data and the roboticdata. For example, the comparison result may indicate that the curvatureof the instrument indicated by the shape data is outside the range ofacceptable curvature indicated by the robotic data (e.g., exceeds thehighest acceptable curvature by a specific amount). As another example,the comparison result may indicate that the shape data corresponding toa specific portion of the instrument is not consistent with the torquemeasurements included in the robotic data (e.g., measurements of thetorque applied to the pull wires).

VI. B. 4. iv. Improving Shape Data Using Data Other than Robotic Data

In other embodiments, the shape data comparison module 907 can comparethe shape data to the image data received from the image data store 910,the shape data to the EM data received from the EM data store 920, theshape data to the 3D model data received from the 3D model data store940, the shape data to other data received from other data store(s) 941,and/or any combination of data received from two or more of the datastores 910-941.

For example, the shape data comparison module 907 may determine, basedon the image data received from the image data store 910, an expectedorientation of the instrument (e.g., at or near the distal end of theinstrument). The shape data comparison module 907 may then determinewhether the shape data is inconsistent with the expected orientation ofthe instrument (e.g., the image data indicates that the tip of theinstrument is pointing in a direction parallel to the anatomical lumen,but the shape data indicates that the tip of the instrument is pointingat an inner wall of the anatomical lumen).

In another example, the shape data comparison module 907 may determine,based on the 3D model data received from the 3D model data store 940,that the anatomical lumen in which the instrument is located has a rangeof possible coordinate values. The shape data comparison module 907 maythen determine whether the shape data indicates that the instrument islocated outside the range of possible coordinate values or whether theshape data indicates that the instrument is shaped in a way that wouldnot fit in the anatomical lumen.

In yet another example, the shape data comparison module 907 maydetermine, based on the EM data received from the EM data store 920, aset of coordinate values corresponding to the current location of theinstrument in a reference coordinate system. The shape data comparisonmodule 907 may then determine whether the shape data is inconsistentwith the expected orientation of the instrument (e.g., the set ofcoordinate values indicated by the shape data is different from the setof coordinate values indicated by the EM data, or deviates from the setof coordinate values indicated by the EM data by more than a thresholdamount).

In yet another example, fluoroscopy (X-ray) images can be analyzed by acomputer vision algorithm to extract the silhouette of the instrument,and the shape data comparison module 907 may then determine whether theshape data is inconsistent with the extracted silhouette of theinstrument.

In yet another example, different sensing modalities can be fit into theworking channel 438 and may be connected to work with the system. Thesesensing modalities include radial endobronchial ultrasound (REBUS)probes, multispectral imaging (spectroscopes), tomography imaging(optical coherence tomography, confocal microscopy, two-photonexcitation microscopy, etc.). Using the sensor data generated by thesesensing modalities, the shape data comparison module 907 can determinewhether the shape data is inconsistent with the sensor data.

VI. B. 4. v. Other Examples of Shape Data Comparison

In some embodiments, the shape data comparison module 907 determinesthat a mismatch between the shape data and the robotic data has beendetected for over a threshold amount of time, and outputs an alertindicating that the instrument may be damaged. For example, the shapedata comparison module 907 may determine that the last five comparisonresults output to the shape data adjustment module 908 indicated thatthe shape data was inconsistent with the robotic data, and output analert (e.g., indicating that the instrument may be damaged, stuck, orotherwise malfunctioning).

Although not illustrated in FIG. 10 , the shape data comparison module907 can, additionally or alternatively, compare the shape data to theestimated state data received from the state estimator 980. In somecases, the shape of the sheath may be known (e.g., based onshape-sensing using optical fibers inside the sheath, or robotic datacorresponding to the sheath). In such cases, the shape data comparisonmodule 907 may access the shape data corresponding to the sheathsurrounding the instrument, and compare the shape data of the instrumentto the shape data of the sheath.

In some cases, the shape data comparison module 907 determines that therobotic data has a higher confidence value than the shape data, andbased on the determination, compares the shape data to the robotic data.Alternatively, in some cases, the shape data comparison module 907determines that robotic data has a lower confidence value than the shapedata, and based on the determination, refrains from comparing the shapedata to the robotic data.

For example, at or near the distal end of the instrument, the confidencevalue assigned to the shape data or strain data may be lower than thoseassigned to the robotic data, because as discussed above, strainmeasurements of distal portions of the instrument may rely onshape/position data determined based on strain measurements of proximalportions of the instrument (e.g., closer to the shape detector 452), andany errors in the strain measurements of proximal portions may bemagnified in the strain measurements of distal portions. On the otherhand, at or near the proximal end of the instrument, the confidencevalue assigned to the shape data or strain data may be higher than thoseassigned to the robotic data.

In some embodiments, the shape data is compared to the robotic data ator near the distal end of the instrument and adjusted as needed, but theshape data is not compared to the robotic data at or near the proximalend of the instrument. In other embodiments, the shape data is comparedto the robotic data at or near both the distal end and the proximal endof the instrument and adjusted as needed.

VI. B. 4. vi. Adjusting Shape Data Using Comparison Result

The shape data comparison module 907 outputs the result of thecomparison to the shape data adjustment module 908. The result of thecomparison may indicate which portion of the shape data, if any, doesnot satisfy one or more conditions indicated by the data to which theshape data is compared (e.g., the robotic data). For example, asdiscussed with reference to FIG. 4D, the comparison result may indicatethat the shape data corresponding to a portion of the endoscope 118indicates that the portion exhibits a curvature value that is notconsistent with the robotic data. In another example, as discussed withreference to FIG. 4E, the comparison result may indicate that the shapedata corresponding to a portion of the endoscope 118 indicates that theportion is moving at a speed that is not consistent with the roboticdata.

In other cases, the comparison result may indicate that the direction ofthe instrument tip indicated by the shape data deviates from thedirection of the instrument tip indicated by the robotic data by morethan a threshold amount, that the shape data indicates that the shape ofthe instrument is such that a portion of the instrument would be outsidethe anatomical lumen, or that the location of the instrument indicatedby the shape data deviates from the location of the instrument indicatedby the robotic data by more than a threshold amount. The comparisonresult may indicate any error or deviation from what the system expectsbased on one or more of the data from various sources and/or estimatedstates from the state estimator 980.

Based on the received comparison result, the shape data adjustmentmodule 908 adjusts the shape data and outputs the adjusted shape data tothe shape data store 902. For example, upon determining that thecurvature value indicated by the shape data is not consistent with therobotic data, the shape data adjustment module 908 may modify the shapedata such that the curvature value is within the acceptable curvaturerange indicated by the robotic data. As another example, upondetermining that the current speed indicated by the shape data is notconsistent with the robotic data, the shape data adjustment module 908may modify the shape data such that the current speed is within theacceptable speed range indicated by the robotic data. In yet anotherexample, upon determining that a portion or all of the shape data doesnot satisfy one or more conditions indicated by the robotic data, theshape data adjustment module 908 may, instead of adjusting the shapedata, discard such shape data. In yet another example, upon determiningthat a portion or all of the shape data does not satisfy one or moreconditions indicated by the robotic data, the shape data adjustmentmodule 908 may, instead of adjusting the shape data, reduce theconfidence in the shape data (e.g., by decreasing the confidence valueassociated with the shape data). The adjusted shape data is stored inthe shape data store 902. In some cases, the adjusted shape data isstored in another data store different from the shape data store 902.

In some cases, the shape data adjustment module 908 may make alternativeor additional adjustments based on other factors. For example, the shapedata adjustment module 908 may adjust the shape data (or adjust theconfidence in the shape data) based on a change in temperature. In suchan example, the shape data adjustment module 908 may adjust the shapedata (or adjust the confidence in the shape data) based on thermalexpansion and contraction properties of the optical fibers. The shapedata adjustment module 908 may make such an adjustment in response todetermining that the received comparison result indicates that the shapedata is at odds with at least one other data. In other cases, the shapedata determination module 906 takes the current temperature into accountwhen determining the shape data based on the received strain data, andthe shape data adjustment module 908 does not make additionaltemperature-based adjustments to the shape data.

In some embodiments, the shape data adjustment module 908 may adjust theshape data (or adjust the confidence in the shape data) based on whetherthe tip of the instrument is being articulated or not. Alternatively oradditionally, the shape data adjustment module 908 may adjust the shapedata (or adjust the confidence in the shape data) based on whethernon-shape-changing strain (e.g., temperature, articulation mode, etc.)is being applied to the instrument. The shape data adjustment module 908may make one or both of these adjustments in response to determiningthat the received comparison result indicates that the shape data is atodds with at least one other data.

FIG. 11 shows an example shape-based state estimation module that may beincluded in the strain-based algorithm module 945. As shown in FIG. 11 ,the shape-based state estimation module 909 receives the adjusted shapedata from the shape data store 902 and determines shape-based estimatedstate data based on the adjusted shape data and prior estimated statedata received from the estimated state data store 985. The shape-basedstate estimation module 909 outputs the shape-based estimated state datato the estimated state data store 985. This process can be repeated togenerate estimated state data for future timestamps. In some cases, theshape-based state estimation module 909 determines an estimated state ofa sheath covering the instrument based on the estimated state of theinstrument.

VII. A. Overview of Shape Data Adjustment Based on Robotic Data

FIG. 12 is a flowchart illustrating an example method operable by asurgical robotic system, or component(s) thereof, for determining andadjusting shape data based on other data available to the surgicalrobotic system, such as robotic data, according to one embodiment. Forexample, the steps of method 1200 illustrated in FIG. 12 may beperformed by processor(s) and/or other component(s) of a medical roboticsystem (e.g., surgical robotic system 500) or associated system(s)(e.g., the strain-based algorithm module 945 of the navigationconfiguration system 900). For convenience, the method 1200 is describedas performed by the surgical robotic system, also referred to simply asthe “system” in connection with the description of the method 1200.

The method 1200 begins at block 1201. At block 1205, the system accessesrobotic data regarding an instrument navigated (or to be navigated)within an interior region of a body. The robotic data can include datarelated to the control of the instrument (e.g., endoscope 118 and/or itssheath), and/or physical movement of the instrument or part of theinstrument (e.g., the instrument tip or sheath) within the tubularnetwork. As described above, the robotic data can include command data,force and distance data, mechanical model data, kinematic model data,and the like.

At block 1210, the system accesses strain data from an optical fiberpositioned within the instrument. The strain data may be indicative of astrain on a portion of the instrument positioned within the interiorregion of the body. In some cases, the strain data indicates one or bothof a strain on the distal end of the instrument and a strain on theproximal end of the instrument. The strain data may be generated by theshape detector 452 and stored in the strain data store 901, and thesystem may access the strain data from the strain data store 901.

At block 1215, the system determines shape data based on the straindata. For example, based on the strain on a specific portion of theinstrument indicated by the strain data, the system may predict theshape of the specific portion of the instrument. The shape data mayinclude angles, coordinates, or a combination thereof indicative of thecurrent shape of the instrument. In some cases, the shape data mayinclude curvature information (e.g., curvature value of one or moreportion of the instrument), orientation information (e.g., roll, pitch,and/or yaw of one or more portions of the instrument), positioninformation (e.g., position of one or more portions of the instrument ina reference coordinate system, which is, for example, used by the systemto navigate the instrument), and/or other information that can be usedto indicate the shape of the instrument.

At block 1220, the system compares the robotic data and the shape data.In some embodiments, the comparison includes determining whether aspecific value included in the shape data satisfies a correspondingcondition indicated by the robotic data. For example, the robotic dataaccessed by the system may indicate that the instrument is not capableof being controlled in a way that results in a curvature value greaterthan a maximum curvature value or outside a given range of curvaturevalues. In such an example, the system may determine whether thecurvature value of a given portion of the instrument indicated by theshape data exceeds the maximum curvature value or is outside the givenrange of curvature values indicated by the robotic data for the givenportion of the instrument. In another example, the robotic data accessedby the system may indicate that the instrument is not capable of beingmoved faster than a maximum speed or outside a specific movement range.In such an example, the system may determine whether the movement (e.g.,speed, movement path, or other time history data) of a given portion ofthe instrument indicated by the shape data satisfies a movementcondition (e.g., maximum speed, movement speed range, etc.) indicated bythe robotic data for the given portion of the instrument. A similartechnique can be applied in other cases such that the system candetermine whether any parameter value indicated by the shape datasatisfies a corresponding shape condition (e.g., minimum, maximum,and/or range values that can indicate whether a given parameter value inthe shape data is or is likely to be erroneous) indicated by the roboticdata.

At block 1225, the system adjusts the shape data based on the comparisonof the robotic data and the shape data. In some embodiments, theadjustment includes modifying at least a portion of the shape data suchthat the determination of the estimated state of the instrument (atblock 1230) is based on the modified portion of the shape data. Forexample, upon determining that the curvature value indicated by theshape data exceeds the maximum curvature value indicated by the roboticdata, the system may modify the shape data such that the curvature valueis less than or equal to the maximum curvature value indicated by therobotic data. As another example, upon determining that the currentspeed indicated by the shape data exceeds the maximum speed indicated bythe robotic data, the system may modify the shape data such that thecurrent speed is less than or equal to the maximum speed indicated bythe robotic data. In other embodiments, the adjustment includes removingat least a portion of the shape data such that the determination of theestimated state of the instrument (at block 1230) is not based on theremoved portion of the shape data. For example, upon determining that aportion or all of the shape data does not satisfy one or more conditionsindicated by the robotic data, the system may discard such shape data ordisregard such shape data in the determination of the estimated state atblock 1230.

Adjusting the shape data may also include assigning a confidence valueor weight to the shape data or adjusting such confidence value or weightassigned to the shape data. For example, upon determining that the shapedata satisfies one or more conditions indicated by the robotic data, thesystem may increase the confidence value or weight associated with theshape data. Alternatively, upon determining that the shape data does notsatisfy one or more conditions indicated by the robotic data, the systemmay decrease the confidence value or weight associated with the shapedata.

At block 1230, the system determines an estimated state of theinstrument based on the adjusted shape data. In some cases, the systemmay determine the estimated state of the instrument based on acombination of the adjusted shape data and data from one or more datastores shown in FIG. 8A and/or one or more estimated state data from thestate estimator 980 in FIG. 8A. At block 1235, the system outputs theestimated state of the instrument. The method 1200 ends at block 1240.

VII. B. Overview of Shape Data Adjustment Process

FIG. 13 is a conceptual diagram illustrating an example method operableby a surgical robotic system, or component(s) thereof, for operating aninstrument, according to one embodiment. For example, the steps shown indiagram 1300 illustrated in FIG. 13 may be performed by processor(s)and/or other component(s) of a medical robotic system (e.g., surgicalrobotic system 500) or associated system(s) (e.g., the strain-basedalgorithm module 945 of the navigation configuration system 900). Forconvenience, the processes illustrated in diagram 1300 are described asperformed by the surgical robotic system, also referred to simply as the“system.”

As shown in FIG. 13 , shape data 1305 and robotic data 1310 are fed intothe decision block 1315. At block 1315, the system determines whetherthe shape data 1305 is acceptable in view of the robotic data 1310. Upondetermining that the shape data is not acceptable, the system proceedsto block 1320 and adjusts the shape data. Upon determining that theshape data is acceptable, the system proceeds to block 1325 to drive theinstrument based at least on the shape data (or adjusted shape data)and/or to block 1330 to navigate the instrument based at least on theshape data (or adjusted shape data).

VII. C. Overview of Shape Data Confidence Adjustment Process

FIG. 14 is a conceptual diagram illustrating an example method operableby a surgical robotic system, or component(s) thereof, for operating aninstrument, according to one embodiment. For example, the steps shown indiagram 1400 illustrated in FIG. 14 may be performed by processor(s)and/or other component(s) of a medical robotic system (e.g., surgicalrobotic system 500) or associated system(s) (e.g., the strain-basedalgorithm module 945 of the navigation configuration system 900). Forconvenience, the processes illustrated in diagram 1400 are described asperformed by the surgical robotic system, also referred to simply as the“system.”

As shown in FIG. 14 , the system takes shape data 1405 and robotic data1410 and generates weighted data 1415. The weighted data 1415 may be aweighted sum of the shape data 1405 and the robotic data 1410, where theshape data 1405 and the robotic data 1410 are weighted based on theirrespective confidence values. If the confidence value associated withthe shape data 1405 is higher than that of the robotic data 1410, theshape represented by the weighted data 1415 may be closer to thatrepresented by the shape data 1405. On the other hand, if the confidencevalue associated with the robotic data 1410 is higher than that of theshape data 1405, the shape represented by the weighted data 1415 may becloser to that represented by the robotic data 1410. At block 1420, thesystem determines whether the shape data 1405 is acceptable in view ofthe robotic data 1410. Upon determining that the shape data 1405 is notacceptable, the system proceeds to block 1425 and adjusts the confidencevalue associated with the shape data 1405. The system then proceeds toblock 1430 to drive the instrument based at least on the weighted datareflecting the adjusted confidence value and/or to block 1435 tonavigate the instrument based at least on the weighted data reflectingthe adjusted confidence value. On the other hand, upon determining thatthe shape data 1405 is acceptable, the system proceeds to block 1430 todrive the instrument based at least on the weighted data 1415 and/or toblock 1435 to navigate the instrument based at least on the weighteddata 1415.

As just discussed with respect to FIG. 14 , a number of embodimentsdescribed herein may adjust confidence values to improve navigation orthe control of the medical instrument. For example, in some cases, anavigation system may use the adjusted confidence value to lower theweight given to the strain-based shape data to determine the location ofthe medical instrument relative to anatomy of a patient, as may berepresented by a preoperative model of the patient. As discussed inother portions of this disclosure, a navigation system (see, e.g., FIG.8A) may receive a number of different state estimates of the medicalinstruments from corresponding state estimators and, according to theembodiment shown in FIG. 14 , the navigation system may lower the weightgiven to the state derived from the strain-based shape data based on thecomparison. Depending on the adjusted confidence, the navigation systemmay disregard the state estimate derived from state estimators using thestrain-based shape data or may lower the impact the state estimator hason determining the estimated state of the medical device.

It is to be appreciated that the converse is also possible forembodiments contemplated by this disclosure. That is, if the comparisonbetween the strain-based shape data and the robotic data-based shapedata determines that the two types of data closely matches (as may bedetermined by a threshold amount) then the navigation system mayincrease the confidence or weight of the state estimate derived from thestrain-based shape data.

Similarly described in FIG. 14 , some embodiments may include a controlsystem that controls the drive of the medical instrument based on thecomparison between the strain-based shape data and the robotic data.Such control systems may, based on the comparison, use or disregard thestrain-based shape data when controlling the pose of the medicalinstrument.

While FIGS. 4D, 4E, and 9-14 are described herein with respect torobotic data, in other embodiments, other data can be used instead ofrobotic data or in combination with robotic data. Also, although sometechniques described herein are described with reference to surgicalrobotic systems, in other embodiments, such techniques can be applied tonon-surgical systems such as medical robotic systems and systems forcontrolling an instrument within interior regions of a body that do notinvolve a surgery.

VIII. Implementing Systems and Terminology

Implementations disclosed herein provide systems, methods andapparatuses for detecting physiological noise during navigation of aluminal network.

It should be noted that the terms “couple,” “coupling,” “coupled,” orother variations of the word couple as used herein may indicate eitheran indirect connection or a direct connection. For example, if a firstcomponent is “coupled” to a second component, the first component may beeither indirectly connected to the second component via anothercomponent or directly connected to the second component.

The functions described herein may be stored as one or more instructionson a processor-readable or computer-readable medium. The term“computer-readable medium” refers to any available medium that can beaccessed by a computer or processor. By way of example, and notlimitation, such a medium may comprise random access memory (RAM),read-only memory (ROM), electrically erasable programmable read-onlymemory (EEPROM), flash memory, compact disc read-only memory (CD-ROM) orother optical disk storage may comprise RAM, ROM, EEPROM, flash memory,CD-ROM or other optical disk storage, magnetic disk storage or othermagnetic storage devices, or any other medium that can be used to storedesired program code in the form of instructions or data structures andthat can be accessed by a computer. It should be noted that acomputer-readable medium may be tangible and non-transitory. As usedherein, the term “code” may refer to software, instructions, code ordata that is/are executable by a computing device or processor.

The methods disclosed herein comprise one or more steps or actions forachieving the described method. The method steps and/or actions may beinterchanged with one another without departing from the scope of theclaims. In other words, unless a specific order of steps or actions isrequired for proper operation of the method that is being described, theorder and/or use of specific steps and/or actions may be modifiedwithout departing from the scope of the claims.

As used herein, the term “plurality” denotes two or more. For example, aplurality of components indicates two or more components. The term“determining” encompasses a wide variety of actions and, therefore,“determining” can include calculating, computing, processing, deriving,investigating, looking up (e.g., looking up in a table, a database oranother data structure), ascertaining and the like. Also, “determining”can include receiving (e.g., receiving information), accessing (e.g.,accessing data in a memory) and the like. Also, “determining” caninclude resolving, selecting, choosing, establishing and the like.

The phrase “based on” does not mean “based only on,” unless expresslyspecified otherwise. In other words, the phrase “based on” describesboth “based only on” and “based at least on.”

The previous description of the disclosed implementations is provided toenable any person skilled in the art to make or use the presentinvention. Various modifications to these implementations will bereadily apparent to those skilled in the art, and the generic principlesdefined herein may be applied to other implementations without departingfrom the scope of the invention. For example, it will be appreciatedthat one of ordinary skill in the art will be able to employ a numbercorresponding alternative and equivalent structural details, such asequivalent ways of fastening, mounting, coupling, or engaging toolcomponents, equivalent mechanisms for producing particular actuationmotions, and equivalent mechanisms for delivering electrical energy.Thus, the present invention is not intended to be limited to theimplementations shown herein but is to be accorded the widest scopeconsistent with the principles and novel features disclosed herein.

What is claimed is:
 1. A method of controlling an instrument, the methodcomprising: measuring strain of one or more optical fibers associatedwith an elongate shaft of an instrument; determining a first shapeestimation of the elongate shaft based on the measured strain;determining, based on the measured strain, a strain-indicatedcharacteristic of the elongate shaft with respect to a mechanicalcondition of the elongate shaft; determining an expected range of themechanical condition of the elongate shaft; determining that thestrain-indicated characteristic is outside of the expected range of themechanical condition; and in response to the determination that thestrain-indicated characteristic is outside of the expected range,determining a second shape estimation of the elongate shaft, the secondshape estimation being based to a lesser degree on the measured straincompared to the first shape estimation.
 2. The method of claim 1,wherein the mechanical condition is one of curvature of the elongateshaft or speed of motion of the elongate shaft.
 3. The method of claim1, wherein the expected range of the mechanical condition is based onrobotic command data used to control movement of the instrument.
 4. Themethod of claim 3, wherein the robotic command data indicates acommanded pitch, roll, yaw, and insertion of the instrument.
 5. Themethod of claim 1, wherein the expected range of the mechanicalcondition is based on at least one of: pull force on one or morearticulation pull wires associated with the elongate shaft; or actuationposition of the one or more articulation pull wires.
 6. The method ofclaim 1, wherein the expected range of the mechanical condition is basedon image data received from a camera associated with a distal portion ofthe elongate shaft.
 7. The method of claim 6, further comprisingdetermining that the image data indicates an orientation of the elongateshaft relative to an anatomical lumen, wherein said determining that thestrain-indicated characteristic is outside of the expected range of themechanical condition is based on the indicated orientation of theelongate shaft.
 8. The method of claim 1, wherein said determining thatthe strain-indicated characteristic is outside of the expected range ofthe mechanical condition involves: determining a position of at least aportion of the elongate shaft relative to a virtual model of ananatomical lumen in which at least a portion of the elongate shaft isdisposed; and determining that the position of the at least a portion ofthe elongate shaft is outside of a boundary of the virtual model of theanatomical lumen.
 9. The method of claim 1, wherein said determining thesecond shape estimation of the elongate shaft involves decreasing aconfidence value associated with the measured strain.
 10. The method ofclaim 1, wherein said determining the second shape estimation of theelongate shaft involves adjusting the measured strain to bring thestrain-indicated characteristic within the expected range.
 11. A roboticsystem for controlling an instrument, the system comprising: aninterface configured to receive signals from one or more optical fibersdisposed at least partially within an elongate shaft of an instrument;and control circuitry configured to: determine a measured strain of theone or more optical fibers based on the received signals; determine afirst shape estimation of the elongate shaft based on the measuredstrain; determine, based on the measured strain, a strain-indicatedcharacteristic of the elongate shaft with respect to a mechanicalcondition of the elongate shaft; determine an expected range of themechanical condition of the elongate shaft; determine that thestrain-indicated characteristic is outside of the expected range of themechanical condition; and in response to the determination that thestrain-indicated characteristic is outside of the expected range,determine a second shape estimation of the elongate shaft, the secondshape estimation being based to a lesser degree on the measured straincompared to the first shape estimation.
 12. The robotic system of claim11, wherein the mechanical condition is one of curvature of the elongateshaft or speed of motion of the elongate shaft.
 13. The robotic systemof claim 11, wherein the expected range of the mechanical condition isbased on robotic command data that indicates at least one of a commandedpitch, roll, yaw, or insertion of the instrument.
 14. The robotic systemof claim 11, wherein the expected range of the mechanical condition isbased on at least one of: pull force on one or more articulation pullwires associated with the elongate shaft; or actuation position of theone or more articulation pull wires.
 15. The robotic system of claim 11,wherein: the expected range of the mechanical condition is based onimage data received from a camera associated with a distal portion ofthe elongate shaft; the control circuitry is further configured todetermine that the image data indicates an orientation of the elongateshaft relative to an anatomical lumen; and said determining that thestrain-indicated characteristic is outside of the expected range of themechanical condition is based on the indicated orientation of theelongate shaft.
 16. The robotic system of claim 11, wherein saiddetermining that the strain-indicated characteristic is outside of theexpected range of the mechanical condition involves: determining aposition of at least a portion of the elongate shaft relative to avirtual model of an anatomical lumen in which at least a portion of theelongate shaft is disposed; and determining that the position of the atleast a portion of the elongate shaft is outside of a boundary of thevirtual model of the anatomical lumen.
 17. The robotic system of claim11, wherein said determining the second shape estimation of the elongateshaft involves adjusting the measured strain to bring thestrain-indicated characteristic within the expected range.
 18. Anon-transitory computer-readable storage medium having stored thereoninstructions that, when executed, cause one or more processors of adevice to at least: measure strain of one or more optical fibersassociated with an elongate shaft of an instrument; determine a firstshape estimation of the elongate shaft based on the measured strain;determine, based on the measured strain, a strain-indicatedcharacteristic of the elongate shaft with respect to a mechanicalcondition of the elongate shaft; determine an expected range of themechanical condition of the elongate shaft; determine that thestrain-indicated characteristic is outside of the expected range of themechanical condition; and in response to the determination that thestrain-indicated characteristic is outside of the expected range,determining a second shape estimation of the elongate shaft, the secondshape estimation being based to a lesser degree on the measured straincompared to the first shape estimation.
 19. The non-transitorycomputer-readable storage medium of claim 18, wherein the expected rangeof the mechanical condition is based on at least one of: pull force onone or more articulation pull wires associated with the elongate shaft;or actuation position of the one or more articulation pull wires. 20.The non-transitory computer-readable storage medium of claim 18, whereinsaid determining the second shape estimation of the elongate shaftinvolves adjusting the measured strain to bring the strain-indicatedcharacteristic within the expected range.